Over the past several years there have been increasing demands placed upon graphics subsystems in all variety of hardware. For example, in the general computing area, even traditionally mundane programs, like presentation software, are including animation and other tools that are requiring faster and more complex graphics computation. In addition, the traditional graphics-intense applications like video, photo editing and gaming are growing both in scope and graphics-intensity. Moreover, vertical systems such as gaming and graphics dedicated computing (e.g. Nintendo Gamecube etc.) have accelerated even faster driving competition with the general computing architecture for graphics supremacy.
During this same time period, hardware manufacturers have sought to meet and exceed the growing demand with dedicated graphics processors having ever-increasing capability. Right now, there are several commercially available graphics processing units (GPUs) that are programmable. Programmable GPUs run programs that are generally called fragment programs. The name “fragment” program derives from the fact that the unit of data being operated upon is generally a pixel—i.e. a fragment of an image. The GPUs can run a fragment program on several pixels simultaneously to create a result, which is generally referred to by the name of the buffer in which it resides. GPUs use data input generally called textures, which is analogous to a collection of pixels.
While both programmable and non-programmable GPUs offer enhanced speed for graphics calculations, programmable GPUs differ in that they offer a high measure of flexibility. For example, prior to programmable GPUs, an application programmer might decide between using CPU time to render a more interesting graphic or using the GPU to increase overall application performance at the cost of displaying a less ideal graphic. Programmable GPUs have combined the speed advantage of prior GPUs with a significant measure of flexibility. In practical terms, programmability is an important advantage because it allows programs to use the graphics chip in ways similar to the system microprocessor. By using the GPU this way, the system can generate virtually infinite graphics effects without loading the system CPU. At the same time, as computers migrated toward more visually rich content, more applications required the complex manipulation of graphics.
The increasing flexibility of programmable GPUs coupled with increased demand for a higher level of graphics hardware controllability spurred the development of programming interfaces for the complex manipulation of images. Such programming interfaces include, for example, Open Graphics Library (“OpenGL”) and Core Image. OpenGL is a specification defining a cross-language cross-platform API for writing applications that produce two-dimensional and three-dimensional computer graphics. The interface consists of over 250 different function calls which can be used to draw complex three-dimensional scenes from simple primitives. OpenGL is widely used in CAD, virtual reality, scientific visualisation, information visualisation and video game development.
Aspects of the present invention utilize aspects of U.S. patent application Ser. No. 10/826,762, entitled “High-Level Program Interface for Graphics Operations,” which is incorporated by reference in its entirety, and generally referred to in this application as Core Image. Core Image is a higher level interface to a graphics subsystem which provides an abstraction layer that hides the complexity of graphics hardware while making available this abstraction layer to all applications. Core Image allows a programmer or program to quickly and efficiently apply a filter or effect to a given image. Furthermore, Core Image minimizes the use of memory and computation time, while efficiently dividing work between the CPU and GPU.
Digital imagery is becoming the backbone of many industries, including scientific visualization, photography, visual effects, and post production. The widespread use of digital imagery has resulted in end users who are not programmers requiring the ability to flexibly manipulate digital images. As the complexity of GPUs increases and as more non-programmer end users require the ability to edit multimedia, the ease with which a user can access these programming interfaces and otherwise utilize programmable GPUs gains importance.
While the powerful Core Image framework provides an optimal environment for both expressing and executing image effects, this may require that a developer write code that chains those filters together by creating the connections or create programmatically a CIFilterGraph that expresses a chain of filters.
It is desirable that non-programmer end users be able to easily edit multimedia using a scripting language. This editing may be carried out by leveraging the Core Image framework or any similar framework that may now exist or that will exist in the future (e.g., even OpenGL). A scripting language is a high-level programming language for connecting diverse pre-existing components to accomplish a new related task. Scripting languages typically allow their source codes to be loaded directly. This source code, or script, is evaluated at runtime, either by, most often, interpreting the script with an interpreter program or, less frequently, by compiling the script on-the-fly and executing the compiled script. Conventional scripting languages typically do not expose image editing features and are neither lightweight on the implementation side in the application nor are they easily understood by non-software-engineering users. It is desirable that a scripting language would be lightweight, easy to use by non-programmers, and capable of applying one or more filters to any available image.
Although Photoshop macros are capable of providing different filter parameters for different output media, Photoshop ties these macros to proprietary software which is limited due to lack of hardware acceleration, an expensive application package, and lack of portability to other applications. It is desirable that a scripting language would be able to edit multimedia from any application.
Microsoft's Windows Presentation Foundation coupled with Microsoft Expression Interactive Designer allows the creation of user interface elements that carry display, rendering, and interactivity attributes. These effects are expressed using the Extensible Application markup Language (XAML). XAML is a declarative XML-based language optimized for describing graphically rich visual user interfaces, such as those created by Macromedia Flash. However, these attributes are limited to basic elements like gradients, solid fills, and strokes, and lack the broad editing capacity demanded by non-programmers in image-enhancement intensive environments.
XML User Interface Language (XUL) and User Interface Markup Language (UIML) are other examples of XML-based user interface languages that also lack broad editing capacity. Scalable Vector Graphics (SVG) is another language supporting 2D graphics and scripted behavior which could be used as an XML-based user interface language, but it is designed primarily for web page implementation instead of robust system-level image manipulation.
Another problem is that there is nothing available that allows the user to attach his manipulations to the original image so that the original data remains untouched. Today's image formats are static, so that fixed results are expressed solely on the pixels. Therefore they don't allow the editing of any of the applied manipulations. On the other hand, a programatic recipe would allow the user to edit or virtually edit the image and its manipulations based on its original data without the loss of any of its dynamics due to rendering into a pixel format.